{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TF - 某个词在某一文本中出现的频率\n",
    "IDF - 逆向文本频率\n",
    "\n",
    "TF-IDF用于评估某一个词对所在文本的重要程度，其在当前文本中出现频次越多，在其他文本中出现频次越少，可认为此词越重要。\n",
    "\n",
    "##### TF计算公式\n",
    "\n",
    "$$\n",
    "TF_{i,j}=\\frac{n_{i,j}}{\\sum_k{n_{k,j}}}\n",
    "$$\n",
    "\n",
    "其中：\n",
    "\n",
    "$$\n",
    "TF_{i,j}\\,\\,-\\,\\,\\text{第}j\\text{篇文本中第}i\\text{个词的}TF\\text{值}\n",
    "$$\n",
    "\n",
    "$$\n",
    "n_{i,j}\\,\\,-\\,\\,\\text{第}j\\text{篇文本中第}i\\text{个词的出现频次}\n",
    "$$\n",
    "\n",
    "\n",
    "$$\n",
    "\\sum_k{n_{k,j}\\,\\,-\\,\\,}\\text{第}j\\text{篇文本中所有词的出现频次}\n",
    "$$\n",
    "\n",
    "$$\n",
    "k\\,\\,-\\,\\,\\text{第}j\\text{篇文本中的词汇量}\n",
    "$$\n",
    "\n",
    "##### IDF计算公式\n",
    "\n",
    "$$\n",
    "IDF_{i,j}\\,\\,=\\,\\,\\log \\frac{\\left| D \\right|}{\\left| D_{t_i} \\right|}\n",
    "$$\n",
    "\n",
    "其中：\n",
    "\n",
    "$$\n",
    "IDF_{i,j}\\,\\,-\\,\\,\\text{第}j\\text{篇文本中第}i\\text{个词的}IDF\\text{值}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\left| D \\right|\\,\\,-\\,\\,\\text{数据库中文本总数}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\left| D_{t_i} \\right|\\,\\,-\\,\\,\\text{数据库中含词}t_i\\text{的文本数量}\n",
    "$$\n",
    "\n",
    "##### 第j篇文本中第i个词的TF-IDF值\n",
    "\n",
    "$$\n",
    "TF\\_IDF_{i,j}\\ =\\ TF_{i,j}\\times IDF_{i,j}\n",
    "$$\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "你 站 在 桥上 看 风景\n",
      "看 风景 的 人 在 楼上 看 你\n",
      "明月 装饰 了 你 的 窗子\n",
      "你 装饰 了 别人 的 梦 \n",
      "\n",
      "{'你': 0, '在': 1, '桥上': 2, '看': 3, '站': 4, '风景': 5, '人': 6, '楼上': 7, '的': 8, '了': 9, '明月': 10, '窗子': 11, '装饰': 12, '别人': 13, '梦': 14} \n",
      "\n",
      "[[(0, 1), (1, 1), (2, 1), (3, 1), (4, 1), (5, 1)], [(0, 1), (1, 1), (3, 2), (5, 1), (6, 1), (7, 1), (8, 1)], [(0, 1), (8, 1), (9, 1), (10, 1), (11, 1), (12, 1)], [(0, 1), (8, 1), (9, 1), (12, 1), (13, 1), (14, 1)]] \n",
      "\n",
      "[(1, 0.30151134457776363), (2, 0.6030226891555273), (3, 0.30151134457776363), (4, 0.6030226891555273), (5, 0.30151134457776363)]\n",
      "[(1, 0.2656320682560318), (3, 0.5312641365120636), (5, 0.2656320682560318), (6, 0.5312641365120636), (7, 0.5312641365120636), (8, 0.11024726933725056)]\n",
      "[(8, 0.1301303789000364), (9, 0.31353884679371596), (10, 0.6270776935874319), (11, 0.6270776935874319), (12, 0.31353884679371596)]\n",
      "[(8, 0.1301303789000364), (9, 0.31353884679371596), (12, 0.31353884679371596), (13, 0.6270776935874319), (14, 0.6270776935874319)]\n"
     ]
    }
   ],
   "source": [
    "from gensim.models import TfidfModel\n",
    "from gensim.corpora import Dictionary\n",
    "import jieba\n",
    "import math\n",
    "\n",
    "raw_texts = [\n",
    "    '你站在桥上看风景',\n",
    "    '看风景的人在楼上看你',\n",
    "    '明月装饰了你的窗子',\n",
    "    '你装饰了别人的梦'\n",
    "]\n",
    "\n",
    "# 对当前文本进行分词\n",
    "texts_list = [[word for word in jieba.cut(text, cut_all=True)] for text in raw_texts]\n",
    "\n",
    "# 输出分词后的文本\n",
    "texts = [' '.join(jieba.lcut(text, cut_all=True)) for text in raw_texts]\n",
    "print('\\n'.join(texts), '\\n')\n",
    "\n",
    "# 构建词典\n",
    "dictionary = Dictionary(texts_list)\n",
    "print(dictionary.token2id, '\\n')\n",
    "\n",
    "# 对文本进行词袋表示\n",
    "bow_texts = [dictionary.doc2bow(text) for text in texts_list]\n",
    "print(bow_texts, '\\n')\n",
    "\n",
    "# 对文本进行TF-IDF表示\n",
    "tfidf = TfidfModel(bow_texts)\n",
    "for text in bow_texts:\n",
    "    print(tfidf[text])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{0: 0.0, 1: 1.0, 2: 2.0, 3: 1.0, 4: 2.0, 5: 1.0, 6: 2.0, 7: 2.0, 8: 0.4150374992788437, 9: 1.0, 10: 2.0, 11: 2.0, 12: 1.0, 13: 2.0, 14: 2.0}\n",
      "[3.3166247903554, 3.342492501982261, 3.189397454976039, 3.189397454976039]\n",
      "[[0.0, 0.30151134457776363, 0.6030226891555273, 0.30151134457776363, 0.6030226891555273, 0.30151134457776363], [0.0, 0.29917793365488515, 0.5983558673097703, 0.29917793365488515, 0.5983558673097703, 0.5983558673097703, 0.12417006142353534], [0.0, 0.1301303789000364, 0.31353884679371596, 0.6270776935874319, 0.6270776935874319, 0.31353884679371596], [0.0, 0.1301303789000364, 0.31353884679371596, 0.31353884679371596, 0.6270776935874319, 0.6270776935874319]]\n"
     ]
    }
   ],
   "source": [
    "import math\n",
    "\n",
    "# 求解切分后文本的TF-IDF\n",
    "def calc_tfidf():\n",
    "    tfidf_list = []\n",
    "    idf_dict = {}\n",
    "    docs_len = len(bow_texts)\n",
    "    token2id = dictionary.token2id\n",
    "    l2_list = []\n",
    "\n",
    "    # 计算文本IDF\n",
    "    for i in bow_texts:\n",
    "        temp_idf_dict = {}\n",
    "        words = 0\n",
    "        for j in i:\n",
    "            words += j[1]\n",
    "            if temp_idf_dict.get(j[0], 0):\n",
    "                continue\n",
    "            temp_idf_dict[j[0]] = 1\n",
    "        for key in temp_idf_dict:\n",
    "            idf_dict[key] = idf_dict.get(key, 0) + temp_idf_dict[key]\n",
    "    for key in idf_dict:\n",
    "        idf_dict[key] = math.log(docs_len / idf_dict[key], 2)\n",
    "    \n",
    "    print(idf_dict)\n",
    "\n",
    "    # 根据计算出的IDF求解L2正则项\n",
    "    for i in bow_texts:\n",
    "        l2_value = 0.0\n",
    "        for j in i:\n",
    "            l2_value += idf_dict[j[0]] ** 2\n",
    "        l2_value = math.sqrt(l2_value)\n",
    "        l2_list.append(l2_value)\n",
    "\n",
    "    print(l2_list)\n",
    "    \n",
    "    # 根据计算TF-IDF\n",
    "    for i in range(0, docs_len, 1):\n",
    "        temp_tfidf_list = []\n",
    "        for j in bow_texts[i]:\n",
    "            temp_tfidf_list.append(j[1] * idf_dict[j[0]] / l2_list[i])\n",
    "        tfidf_list.append(temp_tfidf_list)\n",
    "    \n",
    "    print(tfidf_list)\n",
    "\n",
    "calc_tfidf()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.13 ('py3.8')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "7b6a967c1188be38ccbd2242a0bbc2fce6535f52faf007e2fa9761e3184cc0f8"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
